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ABSTRACT 
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tests do not measure the outcomes of the high school curriculum and hence do 
not reflect student learning in secondary school. Proponents counter that 
while the SAT-I tests are imperfect predictors, they perform a useful role in 
selecting applicants who have a strong likelihood of college success. This 
paper discusses the policy background of this debate and compares criterion- 
related validity evidence for the SAT-I and SAT-II tests. The study used data 
from applicant and enrolled student records at a large, highly selective 
research university for approximately 18,000 first-time freshmen. The 
findings suggest that although the SAT-II tests show stronger criterion- 
related validity than the SAT-I tests, the differences are modest. It is also 
found that the predictive validity of the SAT-I mathematics test improves for 
students from lower income levels and socioeconomic backgrounds. In addition, 
data from the third achievement or subject test of the SAT-II indicate a low 
correlation of scores in this test with freshman year grade point average 
(GPA) and high school GPA. This suggests that the third achievement test may 
be less of a curricular measure, or of less value, in accounting for variance 
in first-year performance. (Contains 7 tables and 11 references.) 
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Abstract 

The University of California is engaged in the elimination of the SAT-I Verbal and 
Mathematics tests as a requirement for freshman admission. Opponents of the SAT-I 
argued that the tests do not measure the outcomes of the high school curriculum and 
hence did not reflect student learning in secondary school. Proponents counter that while 
the SAT-I tests are imperfect predictors, they perform a useful role in selecting applicants 
who have a strong likelihood of college success. This paper discusses the policy 
background of this debate, and compares criterion-related validity evidence for the SAT I 
and SAT II tests. The findings suggest that although the SAT II tests show stronger 
criterion-related validity than the SAT I tests; the differences are modest. It was also 
found that the predictive validity of the SAT I math test improves for students from lower 
income levels and socio-economic backgrounds. In addition, data from the third 
achievement or subject test of the SAT II indicate a low correlation of scores on this test 
with freshman year GPA and high school GPA. This suggests that the third achievement 
may be less of a curricular measure or of less value in accounting for variance in first- 
year performance. 







Background 

In 2001, the President of the University of California proposed eliminating the use 
of the SAT-I Verbal and Quantitative tests for use in freshman admissions decisions. He 
and other critics of the SAT-I tests argued that no longer requiring the SAT I and using ”a 
more relevant standardized test will help strengthen our high schools, focus students' 
attention on mastering subject matter, and create a stronger connection between our 
children's accomplishments in school and their likelihood of succeeding in college.” 
(Atkinson, 2001). This particular criticism of the SAT-I rests on the argument that the 
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test is not based on a curriculum of study, and hence neither measures achievement nor 
hard work in school. Thus critics contend, many students who have excelled in high 
school often doubt their abilities simply because they did not score well or as well as 
expected on the SAT I. Some critics contend that the SAT I has been improperly used as 
a measurement of high school achievement, thus it has a deleterious and biased effect on 
access to the more selective institutions of higher education (Colvin, 1997). 

Perspectives 

Some scholars have found however, that the SAT I does reflect achievement, and 
is a useful tool both for|neasuring what a student has learned in school, but also 
adequately portends their future success in college (Bridgeman, McCamley- Jenkins, and 
Ervin, 2000). They further suggest that the criticisms of the SAT are based on an 
inaccurate interpretation of predictive validity. They suggest that the tests themselves are 
merely a tool used in order to make inferences. If institutions misuse the tool by 
employing it in the wrong situation, it is not the tool that is defective, but rather the 
inferences made by the user of the tool. Proponents of the SAT I point to further research 
that supports the predictive validity value of the SAT-I for college GPA and college 
graduation (Burton and Ramist, 2001, Widaman, 1998). 

The SAT, along with other standardized tests used as part of the college 
admissions process, has long been the focus of criticism throughout the years. In recent 
years, criticism of the SAT and standardized tests in general used for college admissions 
decisions has focused on the argument that the use of such tests in admissions decisions 
restricts the access of students historically under-represented in US higher education 
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(Gandara & Lopez, in press; Lemann, 1999). However, others contend that differences in 
performance on the SAT may be indicative of pervasive ethnic and socioeconomic 
disparities in educational achievement and opportunity (Zwick, 2001). 

Another area of controversy focuses on the use of the third achievement or subject 

A 

area test of the SAT-11. Recently, the university began assigning less weight to SAT-1, 
and more to the SAT-11, also known as the achievement tests. Applicants must take the 
SAT-11 in math and writing, plus a third in any subject they choose. The third 
achievement test is generally viewed as an opportunity for the applicant to demonstrate 
an area of focused study or high ability. For example, many bilingual or students for 
whom English is not their first language choose to take the Spanish, Chinese, or Korean 
language and listening tests. Some have expressed concern that this gives an edge or 
advantage to bi-lingual students who may achieve very high scores on the language test 
without ever having taken a language course in high school. This “bilingual edge” is 
becoming another focus of debate in the continuing saga of equity in college admissions 
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in California (Golden, 2001; Colvin, (1997). According to The College Board, the 
language tests are designed to assess language competency for native English speakers 
who have studied a foreign language in high school. Thus do the debates on the merit 
and fairness of the SAT-1 and SAT-11 achievement tests continue. 

Theoretical Framework 




Although many of the contemporary criticisms of the SAT have been around for 
several years, most recently critics have voiced the concern that the SAT tests are not 
properly aligned with the high school curriculum and thus do not reflect what a student 
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has studied in school. These arguments are based on the notion of a correspondence or 
alignment between what a student studies in school, and the assessment test used to 
determine or assess learning of the subject matter. Pertinent to an understanding of the 
relationship between predictor and criterion is Point — to-Point theory (Asher & 

Scriarrino, 1974). This theory suggests that predictive validity is enhanced when the 
predictor shares more common space with the criterion. Point-to-Point theory suggests 
that the use of scores on a standardized placement tests to predict the normative outcome 
variable of course grade or overall GPA will generally result in low to modest predictive 
validity coefficients and hence diminished predictive validity. The notion that the SAT-II 
subject tests are more highly correlated with the freshman grades is based, at least 
implicitly, on the premise that the SAT-II tests are more closely aligned with the 
secondary school curriculum. A current critique of the SAT focuses on what some 
believe is a lack of correspondence or overlap between what is taught in school, and what 
is measured on the tests (Lemann, 1999). SAT critics maintain that if we increase the 
correspondence between what a student studies in high school and what is measured on 
standardized tests of achievement, then predictive validity will improve. 

Predictive validity is essentially an attempt to approximate the future in the 
present. Thus to the extent that current assessment tests mirror the future demands of 
college courses, predictive validity coefficients should be enhanced. Based on the 
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perceived closer alignment of the SAT-II subject area tests with the high school 
curriculum, contemporary critics of the use of the SAT-I at the University of California 
contend that the SAT-II represents a much fairer and accurate assessment of what 
students learn in school, and therefore should replace the SAT-I. The SAT-I is viewed by 
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some critics as primarily a measure of aptitude rather than achievement (Lemann, 1999; 
Gandara (in press). They argue that the SAT II, which focuses on knowledge of specific 
subject matter such as mathematics, English, history, science, and foreign language, 
measures student accomplishment and begins to approximate an appropriate test for 
university admissions. This study tests this assertion by focusing on the relative 
predictive validity of the SAT-I and SAT-II subtests for explaining variance in first year 
grade point average. To test the assertion that the SAT II is a better measure of what a 
student studied in high school, the correlation coefficients for the SAT II, SAT I and high 
school grade point average are also included and analyzed. 

Research Questions 

Despite these competing points of view, the central question for many of the 
highly selective universities is how well the various standardized assessments predict 
success in college. Therefore the central question guiding this inquiry focuses on the 
comparative value of ea“ch of the various standardized assessment tests in predicting 
college success. The criterion or dependent variable for college success was first year 
grade point average. 

There are three primary research questions that guided this study. 

1. How well do commonly used standardized college admission tests explain 
variance in freshman grades? 

2. Do tests of “curriculum” demonstrate greater predictive validity than tests 
of “aptitude” in explaining variance in freshman student performance? 
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What is the relation of the third achievement test of the SAT-II to high 
school and first year college grades? 



Data Sources 

Using data obtained from applicant and enrolled student records in a large, highly 
selective research university, this paper presents evidence to test the hypothesis that the 
SAT-II is of greater predictive validity than the SAT I. Part one of the study used six 
years of test score data merged with institutional records that included transcripts of 
college courses and grades earned for the fall, 1 996-200 1 entering freshman cohorts. The 
merging of these data enabled the investigators to determine the criterion-related validity 
evidence of the SAT I and SAT II entrance exams with respect to first year grade point 
average. Part two of the study gathered and analyzed evidence to test the hypothesis that 
scores on the SAT II tests would demonstrate greater criterion-related predictive validity 
with freshman year performance compared with the SAT I. 

Part two of the study provides data on the correlation coefficients used to test the 
argument that the SAT II is more reflective of what a student has studied in school. 
Guided by the theoretical constructs provided by Point-to-Point theory, data were 
analyzed to determine the relation between the various SAT II subtests, particularly the 
third achievement test, and performance in college. Additional data were examined to 
determine if significant differences in the amount of variance accounted for in GPA could 
be found for students from different socio-economic backgrounds on both the SAT-I and 
SAT-II. Findings were compared with the predictive validity found for the SAT I on the 
same outcome measures. 
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The Relative Contribution of High School Grades, SAT I and SAT II Scores in 
Predicting Success at UCSD 

Part I of this research project provides evidence on the relative contribution of 
high school grade point average, SAT I and SAT II scores in predicting success at UCSD 
for first-time freshmen. 

Methods 

The official high school grade point average, SAT I, and SAT II scores, of 
approximately 1 8,000 first-time freshmen who enrolled at the University of California, 
San Diego (UCSD) between fall 1996 and Fall 2001 were used for this study. High 
school grade point average is an uncapped, honors-weighted, GPA measured on a scale 
ranging from 0.0 to 4.0; however, due to honors courses HSGPA may exceed 4.0. The 
SAT I is a combination of the students’ verbal and math scores. The SAT II is a 
combination of the students’ English and math scores as well as a score on a third 
achievement test of the students’ choosing. 

Hierarchical regression analysis was used for this predictive validity study. One 
of the major uses of regression analysis in predictive research is for the selection of 
applicants, whether that be for college, a job or some training program. The regression 
equation is used so that the basis of an applicant’s status on a set of predictors may be 
used to predict his or her performance on some criterion (Pedhazur, 1997, p. 138). High 
school GPA, and composite SAT I and SAT II scores served as the predictor variables to 
measure the amount of variance accounted for in predicting college success. Success in 
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college was defined as a students’ spring grade point average as measured at the end of 
their first academic year attending UCSD. The effects of high school grade point 
average, SAT I and SAT II were analyzed independently and in combination with each 
other. Predictor variables were entered into each regression analysis in the order 
displayed in Table 1 (e.g., for the 7‘'’ regression analysis, HSGPA was entered into the 
equation first followed by SAT I and then SAT II) for each given year. 

Results 

Results from each regression analysis are displayed in Table 1 (see appendix). 
The table displays the explained variance in first-year UCGPA that is accounted for by 
the three predictor variables. Explained variance, also referred to as the coefficient of 
determination or represents the proportion of total variance in the criterion or 
outcome variable (e.g., UCGPA in this study) that is explained or accounted for by a 
predictor variable (e.g., HSGPA, SAT I and SAT II). For this study the three predictor 
variables included high school GPA, composite SAT I and SAT II scores for all first-time 
freshmen entering UCSD between fall 1996 and fall 2001. 

Looking at each predictor variable as the sole predictor in the model, as shown in 
Table 1, HSGPA accounts for the greatest amount of variability in all five years and was 
the single best predictor^for the pooled data (R^ = 0.1 13). The next single best predictor 
was the SAT II composite scores, which accounted for the second largest amount of 
variability for all five years in with respect to the pooled-data (R^ = 0.097). Finally, SAT 
I composite scores were ranked third accounting for only 6.5% of the variance in 
UCGPA. 
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With respect to the models including a combination of predictor variables, rows 
(4) through (7) in Table 1, HSGPA and SAT II scores account for the greatest amount of 
variability (R^ = 0.192) in predicting UCGPA, beyond any single predictor variable for 
all five years and with the pooled data. The next best set of predictors includes HSGPA 
and SAT I accounting for 17% of the variability with the pooled data. When only using 
SAT I and SAT II scores in the model, the proportion of explained variance by SAT I 
scores is not significantly different from zero. This result could be due to the 
multicollinearity between SAT I and SAT II scores. Multicollinearity may lead to 
difficulties in the estimation of regression statistics. The least ambiguous definition of 
multicollinearity is that it refers to the absence of orthogonality in a set of independent 
(predictor) variables. When two variables are orthogonal, they are independent of each 
other and the correlation between them is zero. Multicollinearity only refers to the 
interrelations among the independent/predictor variables only (Pedhazur, p. 233). 

Finally, when including, HSGPA, SAT II and SAT I scores into the equation, 
SAT I does not increase the amount of variability accounted for in predicting UCGPA 
(R^ = 0.192) after the effects of HSGPA and SAT II scores are taken into account. 
Adding SAT I scores into the equation does not add any incremental power in predicting 
UCGPA. Again, this may be due to the multicollinearity between SAT I and SAT II 
scores. 



Overall the best model in predicting first-time freshman GPA includes HSGPA 
and SAT II scores, which accounted for 19% of the variability, approximately one-fifth 
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of the total variance in UCGPA. In accounting for only one-fifth of the variance, roughly 
four-fifths is left unexplained. This relatively low level of predictive power tends to be 
the norm in admissions research. One of the known reasons for the low predictive power 
of HSGPA and SAT scores, particularly at UCSD is a restriction of range problem. 
Students with low test scores and grades often do not apply to selective universities such 
as UCSD. Of the students who do apply, only those with relatively high grades and test 
scores tend to be admitted. As a result almost all admitted students have high grades and 
test scores leaving little variability and a limited range with which to fully assess the 
predictive validity of these admission criteria. 

Discussion 

A review of the results suggests three main conclusions. First, high school grade 
point average was the single best predictor of success as measured by first-year GPA at 
UCSD. SAT II was the second best predictor of success. 

Secondly, using both test scores and grades in combination increases the 
proportion of explained variance beyond that which is possible with a single variable. Of 
the SAT I or SAT II, which is the better predictor when coupled with high school grades? 
Results showed that SAT II scores and high school grades explain the greatest amount of 
variability in predicting first-year grades, compared with the SAT I and high school 
grades. 

Additionally, it was found that SAT I scores do not significantly increase the 
amount of explained variability in predicting success when SAT II scores and high school 



grades are also included in the prediction equation. Adding SAT I into the equation does 
not improve the prediction. 

Summary of Results 

• High school gra3e point average was the single best predictor for three of the five 
years studied (1997, 1998 & 2000) and the single best predictor for the pooled 
data. 

• SAT II was the single next best predictor for two of the five years studied (1996 
& 1999). 

• SAT II scores and HSGPA account for a greater proportion of variance in 
predicting UCSD freshmen grades, compared to SAT I scores and HSGPA across 
all years studied. 

• The three predictors combined (HSGPA, SAT I and SAT II), account for the 
greatest amount of variance in predicting UCSD freshmen grades; however, for 
three of the five years studied, the amount of variability accounted for by SAT I 
scores was not found to be significantly different from zero (1996, 1997 & 1998). 

• SAT I scores add very little to the prediction equation in predicting UCSD 
freshmen grades when HSGPA and SAT II scores are also entered into the 
equation. 

• Overall, HSGPA and SAT II scores account for the greatest amount of variability 
(19%) in predicting UCSD freshmen grades in the pooled, 5-year data. 

• Adding SAT I into the equation does not improve the prediction with the pooled 



HSGPA and SAT II only account for approximately one-fifth of the total variance in 
UCSD freshmen grades while four-fifths is still unexplained. (This could be due to a 
restriction of range problem found with selective admissions practices at research 
universities. Students with low GPAs and test scores tend not to apply to selective 
universities, and the grades and test scores of the admitted group are further truncated 
through the selective admissions process. 

The Relation of SAT I and SAT II sub-tests to High School and College Grades 

The second set of analyses examined the inter-correlation among the various SAT 
I and SAT II subtests arid the coefficients of these subtests with high school and first year 
grades. Correlation matrices were produced for different student groupings to note 
differences between demographic sub-groupings such as first generation college students, 
students from high vs. low performing high schools, income level, and first language 
learned at home. 



Method 



This part of the investigation focused on the correlation coefficients derived from 
scores of first time students on the SAT I and SAT II subtests and the official high school 
GPA and first year college GPA of this entering cohort. Pearson Product-Moment 
correlation coefficients Vere obtained and compared to estimate the inter-correlations of 
these measures, and to compare the relationship of the SAT I and the SAT II to high 
school and college GPA’s. The correlation matrices include coefficients among the SAT 
I and SAT II subtest scores, and the official high school GPA and first year college GPA 
of the fall, 2001 freshman class. This part of the analysis was intended to provide 
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evidence for the research question that focused on the comparative criterion-related 
validity of aptitude versus curriculum-based measures of student knowledge. The 
correlation matrices also include the coefficients for the SAT II third achievement test, 
and high school and college GPA’s. These data were included to provide data for 
research question three of this investigation that focused on the criterion-related validity 
evidence of the third achievement test with respect to this exam’s value both as a measure 
of student effort within their high school curriculum and a predictor of first year 
performance. 

Results 

The correlation coefficients derived for the entire cohort of the first time entering 
freshmen cohort are presented in table 2. Included in this table are descriptive data that 
include the means and standard deviations for the various independent and dependent 
variables. The descriptive data for both the SAT I and SAT II subtests indicate that the 
mean scores on the math tests and the third achievement test were higher than the scores 

o 

on the English and verbal portions of the exams. As was found in part one of this 
investigation, high school GPA demonstrates the strongest relationship with first year 
college GPA (.299 or about 9% of the variance). The SAT II English and SAT I Verbal 
tests show a stronger relationship with first year GPA than the SAT I and SAT II 
mathematics tests. However, with respect to high school GPA, the coefficients derived 
from the respective math test scores are approximately the same as those found for the 
English and verbal subtests. Although students tended to achieve relatively high scores 
on the third achievement test (m=614, sd.=109), with the exception of the SAT I math 



test, the predictive validity coefficients suggest that this accounts for the least amount of 
the variance in both high school and college GPA compared with the other measures. 



Analysis by Student Sub-Grouping 

To determine if the predictive validity coefficients between the SAT I and SAT II 
exams and high school and college GPA varied for different demographic groupings, 
separate analyses were conducted using student sub-populations. These sub-populations 
included students disaggregated into the following categories: 



1 . First Generation College Status. This grouping compares the 
performance of students who were the first members of their 
family to complete college with those whose parents had 
completed college. These data are in table 3. 

2. High School Environment This construct groups high schools 
by a combination of school level factors including the breadth of 
the curriculum, socio-economic characteristics, and college 
attendance rates of graduates. Using this score high schools are 
grouped in quintiles for analysis with the highest achieving high 
schools in the first quintile. These data are in tables 4-5. 

3. Income level. Students were grouped by reported household 
income. These data are shown in table 6 

4. First Language Learned. Students were grouped according to the 
first language they reported learning at home. The results of this 
analysis are shown in table 7. 

5. First Generation and Low Income. This grouping includes 
students who were both the first generation in their family to 
complete college and also came from a low-income household. 
Data from this analysis are given in table 8. 



Results from Analysis of Student Sub-Groupings 

The following section describes the results of the analysis for the student sub- 
populations studied in this investigation. 
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First Generation College Status. In general, first generation college students 
tended to achieve lower high school GPA’s and test scores than non-first generation 
students as shown in table 3. In addition, the predictive validity coefficients for SAT I, 
SAT II, and high school^ grades with respect to the dependent variables of high school and 
college GPA were generally lower for first generation college attendees. This was 
particularly true in the verbal and English sub-tests of the SAT. It is interesting to note 
however that the criterion related validity of the SAT I math test for first-generation 
students with respect to college GPA was stronger than the coefficient found for non-first 
generation students. However in both instances the amount of variance explained was 
relatively small. 

High School Environment. Analysis of the correlation coefficients by the 
ranking of high schools did not suggest a particular pattern with respect to the SAT. 
Interestingly, once high school are disaggregated into quintiles according to the 
environmental construct, the SAT I math test appears to be a somewhat stronger predictor 
of first year GPA in lower performing high schools. There is a significant difference in 
the predictive validity coefficient in first quintile high schools (-.001) and fifth quintile 
high schools (.228). Moreover, for the lowest performing high schools, the SAT I math 
score shows a somewhat stronger relationship with college GPA than high school GPA. 

Income Level. Across all income levels, the SAT II tests account for a somewhat 
higher amount of variance than the SAT I tests. The exception to this general pattern is 
found for the third achievement test. Although students from all income levels achieve 
approximately similar scores on this sub-test, the coefficients between both the third 
achievement test, high school, and college GPA’s decrease as income levels drop. 
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However it is interesting to note that as income level drops, the relative strength of the 
coefficients of the SAT I increase, particularly the SAT I math test. This suggests that 
the SAT I math test may have greater predictive validity for low-income students than the 
third achievement test. As shown in table 7, for students in the first generation and low- 
income category, the relative difference between the coefficients for the two exams 
narrows considerably with test scores explaining a greater amount of variance compared 
with non first generation, low-income students. Also, the coefficients between high 
school GPA and first year GPA are lower for students who are both first generation and 
low income. In the case of first generation and low-income students, the observed 
coefficient for first year GPA and the SAT II math test exceeds that of high school GPA. 
The predictive validity coefficient between scores on the third achievement test and first 
year GPA is much lower for the first generation, low income students compared with non 
first generation, low income students. 

Primary Language. The SAT II exams accounted for more variance in first year 
college GPA than the SAT I for all three language groupings as shown in table 6. The 
third achievement test accounts for a smaller proportion of variance in both high school 
and college GPA, although non-native speakers of English have higher scores on this 
exam than the other linguistic groupings. Analysis of the coefficients among the exams 
also indicates that scores on the third achievement test are less related to scores on the 
other SAT subtests when disaggregated by linguistic grouping. 
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Summary and Implications 



Although the SAT-II achievement tests were found to be a better predictor of 
freshman grade point average than the SAT-I, the differences between the predictive 
validity coefficients for the two sets of exams with respect to freshman GPA were 
relatively modest. When analyzed by income level, the relative strength of the 
coefficients for the SAT I increased relative to the SAT II at lower levels of income. A 
similar pattern was noted with respect to the high school rankings and for first generation, 
low-income students. The general exception to this pattern was the predictive validity 
coefficients for the SAT II third achievement test. Although the predictive validity 
coefficients for the third achievement test and college GPA were somewhat lower for the 
total cohort, when analyzed by demographic categories, it was found that the relationship 
between scores on this exam and high school and college GPA was significantly lower 
depending on the high school environment, income level, and native language of the 
student. This finding suggests that the third achievement test for certain groups is not as 
useful in predicting freshman performance or as reflective of high school grades. This 
finding merits further investigation to better understand this relationship. 

Limitations of this Study 

The populations selected for this study were from a single institution, thus the 
findings may be limited to highly selective universities that use standardized admissions 
test in the selection process. Also, as discussed earlier, the competitive nature of 
admissions to the university has resulted in a restriction of range in the distribution of 
high school grades and test scores of admitted and registered students. This truncated 
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distribution has the effect of lowering the predictive validity coefficients. This may lead 
to errors in interpretation of the findings. The relatively low correlation coefficients 
found for the third achievement test, particularly for certain demographic groupings, 
needs further investigation to illuminate the potential lack of correspondence between 
this exam and student high school and college grades. 

Further Research 

Data from this study may be useful to further inform the current debate taking 
place in higher education over the use of standardized tests as adequate measures of 
achievement and academic promise. One area for additional research might include the 
relatively stronger predictive validity of the SAT I math test for students from lower 
socio-economic groupings. Another possible area for future research may be the lower 
level of correspondence between the third achievement test and grades in both high 
school and college. Further research may also provide insights into the development of 
an admissions exam that better reflects the high school curriculum or state high school 
graduation standards and requirements. Improving the correspondence between what 
student studies in high school, and is subsequently measured on exams, may be of greater 
value in evaluating the effectiveness of the high school curriculum, and in selecting 
students for admission to the university. 

The findings from this study also suggest the importance of identifying other 
predictor variables or assessments that demonstrate a relationship to college learning 
outcomes, retention, and persistence. Other indicators included in a comprehensive 
review might include a student’s rank in their high school class, leadership, special 
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talents, writing ability, and perseverance. This research should be continued to further 
understand the relation of certain predictor variables to success at the university while 
admitting students who can enrich the university environment with a variety of skills and 
backgrounds. " 
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